Image processing apparatus, image-pickup apparatus, and image processing method

ABSTRACT

The present invention provides an image processing apparatus capable of obtaining good shake-corrected images in electronic image stablization irrespective of changes of image-taking conditions. An image processing apparatus comprising: a shake correcting part that performs coordinate transformation processing based on shake information to an input image that is generated by use of an image-pickup device; and a method changing part that changes a coordinate transformation method for the coordinate transformation processing.

BACKGROUND OF THE INVENTION

The present invention relates to an image processing technique thatobtains an output image with reduced shake by performing coordinatetransformation processing as shake correction processing to an imagecaptured by an image-pickup apparatus.

Shake correction methods provided in image-pickup apparatuses includeoptical image stabilization and electronic image stabilization. Theelectronic image stabilization selects as a take-out area a region to beactually output (e.g., recorded or displayed) from within one frameimage that is generated by use of an image-pickup device, and moves thetake-out area in accordance with amount of shake for each frame. Then,coordinate transformation processing is applied to each take-out areaimage to thereby output an image with reduced shake (or ashake-corrected image).

Shake of an image-pickup apparatus can be detected by using an angularvelocity sensor or angular acceleration sensor, or by comparing inputimages for two successive frames to detect motion vectors on the imagesas shake.

Shake appearing on an image includes combinations of various types ofshake such as translation and rotation. Besides, characteristics ofapparent shake on an image vary depending on the amount of shake orimage-taking conditions such as zooming position of an imaging opticalsystem. Accordingly, a technique is desired that provides properlyshake-corrected images irrespective of various situations of shakeoccurrence.

For example, Japanese Patent Laid-Open No. 2002-116476 proposes a shakecorrection technique that uses an angular velocity sensor for shakedetection. With this technique, although shake in the direction ofpanning and/or tilting (panning/tilting shake) can be detected,translational shake cannot be detected. Thus, in normal image-taking,shake correction is performed on the assumption that translational shakecan be approximated to shake in the direction of panning or tilting, butin close-up image-taking, shake correction itself is not performedbecause the approximation does not hold in close-up image-taking.

Japanese Patent Laid-Open No. 2004-227003 proposes a shake correctiontechnique that allows selection of an image shake correctioncharacteristic appropriate for various image-taking conditions (i.e., amode in which cut-off frequency for shake correction is high and a modein which the cut-off frequency is low) and/or the range of shakecorrection (i.e., a range in which correction optical devices can move).

However, although the technique proposed in Japanese Patent Laid-OpenNo. 2002-116476 can provide a properly shake-corrected image except inclose-up image-taking, it cannot provide a shake-corrected image inclose-up image-taking because it does not perform shake correctionitself in close-up image-taking. The shake correction technique proposedin Japanese Patent Laid-Open No. 2004-227003 is mainly focused onoptical image stabilization and is not applicable to electronic imagestabilization.

BRIEF SUMMARY OF THE INVENTION

The present invention provides an image processing technique capable ofobtaining good shake-corrected images in electronic image stablizationirrespective of changes of image-taking conditions.

An image processing apparatus of one aspect according to the presentinvention, comprising: a shake correcting part that performs coordinatetransformation processing based on shake information to an input imagethat is generated by use of an image-pickup device; and a methodchanging part that changes a coordinate transformation method for thecoordinate transformation processing.

The image-pickup apparatus that installed the image processing apparatuscompose another aspect according to the present invention.

Further, an image processing apparatus of another aspect according tothe present invention, comprising the steps of: obtaining shakeinformation; performing coordinate transformation processing based onthe shake information to an input image that is generated by use of animage-pickup device; and changing a coordinate transformation method forthe coordinate transformation processing.

Other objects and further features of the present invention will becomereadily apparent from the following description of the preferredembodiments with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the configuration of an image-pickup apparatus (imageprocessing apparatus) as the first embodiment of the invention;

FIG. 2 is a flowchart illustrating shake correction processing in thefirst embodiment;

FIG. 3 illustrates the relation between amount of shake and shakecorrection methods;

FIG. 4 illustrates positional relation between an image-pickup apparatusand image-pickup planes when translational shake is occurring;

FIG. 5 illustrates positional relation between an image-pickup apparatusand image-pickup planes when large panning shake is occurring;

FIG. 6 illustrates positional relation between an image-pickup apparatusand image-pickup planes when small panning shake is occurring;

FIG. 7 illustrates apparent shake that appears on an image when a scenehas different depths;

FIG. 8 illustrates relation of encompassing among shake correctionmethods;

FIG. 9 shows the configuration of an image-pickup apparatus as thesecond embodiment of the invention;

FIG. 10 is a flowchart illustrating shake correction processing in thesecond embodiment;

FIG. 11 illustrates a range in which shake correction is possible whenan output angle of view is large;

FIG. 12 illustrates a range in which shake correction is possible whenan output angle of view is small;

FIG. 13 illustrates relation among amount of shake, the size of anoutput angle of view, and shake correction methods;

FIG. 14 shows the configuration of an image-pickup apparatus as thethird embodiment of the invention;

FIG. 15 is a flowchart illustrating shake correction processing in thethird embodiment;

FIG. 16 illustrates relation between amount of shake and shakecorrection methods;

FIG. 17 shows the configuration of an image-pickup apparatus as thefourth embodiment of the invention;

FIG. 18 is a flowchart illustrating shake correction processing in thefourth embodiment;

FIG. 19 illustrates relation among zoom position, output angle of view,and shake correction methods; and

FIG. 20 generally shows a modification of an embodiment of theinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described belowwith reference to drawings.

Embodiment 1

FIG. 1 shows the configuration of an image-pickup apparatus such as avideo camera and a digital still camera that includes an imageprocessing apparatus as the first embodiment of the present invention.In FIG. 1, reference numeral 101 denotes an optical system that formssubject images, and 102 denotes an image-pickup device such as a CCDsensor and a CMOS sensor that photoelectrically converts subject imagesformed by the optical system 101. Reference numeral 103 denotes an imageforming circuit for generating a video signal from an electrical signaloutput by the image-pickup device 102.

The image forming circuit 103 includes an A/D conversion circuit 104, anautomatic gain control circuit (AGC) 105, and an automatic whitebalancing circuit (AWB) 106, and generates a digital video signal. TheA/D conversion circuit 104 converts an analog signal to a digitalsignal. The AGC 105 provides level correction of a digital signal. TheAWB 106 provides white level correction of images.

Reference numeral 107 denotes frame memory for temporarily storing andholding one or more frames of a video signal generated by the imageforming circuit 103. Reference numeral 108 denotes a memory controlcircuit for controlling video signals that are input to and output fromthe frame memory 107. Reference numeral 109 denotes a shake analysispart for detecting shake of the image-pickup apparatus from neighboringframe images and analyzing trend of the motion, including a motiondetection circuit 110 and a shake analysis circuit 111.

Reference numeral 112 denotes a correction method selection circuit asmethod changing part, which selects a coordinate transformation method(hereinafter referred to as a “shake correction method”) optimal ascoordinate transformation processing to be applied to an input imagebased on amount of shake.

Reference numeral 113 denotes a shake correction circuit that performsshake correction by a shake correction method selected at the correctionmethod selection circuit 112. Reference numeral 114 denotes a videooutput circuit, which serves as an output part that displays ashake-corrected image on a display not shown or records the image on arecording medium such as a semiconductor memory, optical disk, andmagnetic tape.

Reference numeral 100 denotes a main controller for controllingoperations of the image pickup device 102, image forming circuit 103,memory control circuit 108, shake analysis part 109, correction methodselection circuit 112, shake correction circuit 113, and video outputcircuit 114. The main controller 100 may be a CPU and the like.

The operation of the image-pickup apparatus (image processing apparatus)thus configured will be described with respect to the flowchart shown inFIG. 2. The operation described here is carried out in accordance with acomputer program (software) stored in memory not shown of the maincontroller 100. The same applies to other embodiments discussed below.

In FIG. 2, at step S201, a subject image formed by the optical system101 is photoelectrically converted by the image pickup device 102. Theimage pickup device 102 outputs an analog signal corresponding to thebrightness of the subject, which analog signal is input to the imageforming circuit 103. In the image forming circuit 103, the A/Dconversion circuit 104 converts the analog signal to a 14-bit digitalsignal, for example. After further being subjected to signal levelcorrection and white level correction at the AGC 105 and AWB 106, thedigital video signal (i.e., a frame image as an input image) istemporarily recorded and held in the frame memory 107.

In the image-pickup apparatus, frame images are generated in sequence ata predetermined frame rate and frame images stored and held in the framememory 107 are input to the shake analysis part 109 sequentially. Frameimages stored in the frame memory 107 are updated sequentially. Theseoperations are controlled by the memory control circuit 108.

At step S202, shake information including amount and direction of shakeis detected. Specifically, the motion detection circuit 110 detectsmotion vectors that indicate displacement of the subject image and itsdirection between successive frame images. Motion vectors can bedetected using a general detection method such as template matching andgradient methods, although there is no limitation on the method. Inaddition, motion vectors are detected in a plurality of image areas.These motion vectors represent shake information that indicates theamount and direction of apparent shake of individual subjects appearingon an image.

Motion vectors in a plurality of image areas detected by the motiondetection circuit 110 are integrated by the shake analysis circuit 111to generate a representative motion vector that represents shake of theentire image.

Although this embodiment describes a case where amount of shake isdetected by an image processing operation method that uses frame imagesas input images, a shake sensor such as an angular velocity sensor maybe also employed for detection of shake amount.

At step S203, shake occurring on the image which is currently beingtaken is analyzed using amount of shake, which is contained in the shakeinformation provided by the shake analysis part 109, and a shakecorrection method optimal for correcting the shake is selected.

Here, description will be given of how to select a shake correctionmethod in this embodiment. FIG. 3 illustrates the relation betweenamount of shake (i.e., magnitude of shake) and a corresponding shakecorrection method optimal for it. Apparent shake occurring on an imageresults from shake of the image-pickup apparatus itself andcharacteristics of apparent shake occurring on an image vary with themagnitude or type of shake of the image-pickup apparatus itself. Thisembodiment uses the amount of apparent shake occurring on an image toestimate the magnitude of shake of the image-pickup apparatus itself anddetermines a shake correction method of how much degree of freedomshould be applied.

Specifically, the correction method selection circuit 112 selects ashake correction method to be applied by determining a representativemotion vector that represents motion of the entire image that is output,by the shake analysis part 109.

First, if determination of the representative vector shows that amountof shake is small and no shake due to rotation around the optical axisand zooming is occurring on the image, a correction method based ontranslation transformation that is one of coordinate transformationmethods, is selected (hereinafter referred to as “translation correctionmethod”).

Translation correction method is a method that corrects shake on animage by shifting the image in vertical and horizontal directions.

For example, assume that a point on an imagea=[x,y,1]^(T)moves to a pointa′=[x′,y′,1]^(T)due to translational shake:T=[T_(x),T_(y),1]^(T)

In this case, to correct translational shake T, point a′ should betranslated in a direction opposite to the shake:T′=[−T _(x) ,−T _(y),1]^(T)

Thus, translation correction can be expressed as: $\begin{matrix}{a = {a^{\prime} + T^{\prime}}} & ( {{Expression}\quad 1} ) \\{\begin{bmatrix}x \\y \\1\end{bmatrix} = {\begin{bmatrix}x^{\prime} \\y^{\prime} \\1\end{bmatrix} + \begin{bmatrix}{- T_{x}} \\{- T_{y}} \\0\end{bmatrix}}} & ( {{Expression}\quad 2} )\end{matrix}$

The value of translational shake T can be calculated from amount ofshake between successive frame images that is sent from the shakeanalysis part 109.

Points a and a′ and translational shake T are expressed usinghomogeneous coordinates. The same applies to the following descriptionas well.

While it is assumed here that shake can be corrected by translationcorrection method when amount of shake is small, in reality,panning/tilting shake may be occurring in addition to translationalshake. However, when shake of the image-pickup apparatus itself isminute, panning/tilting shake can be approximated to translationalshake.

FIG. 4 illustrates the relation between an image-pickup apparatus thatis having translational shake and displacement of an image-pickup plane.As illustrated in the figure, when translational shake is occurring onthe image-pickup apparatus 401, image-pickup planes 402 and 403 beforeand after the shake move on the same plane to form a common (i.e.,overlapping) area 404. In contrast, as illustrated in FIG. 5, when largepanning/tilting shake occurs on the image-pickup apparatus 501,inclination occurs between image-pickup planes 502 and 503 before andafter the shake. Consequently, a common area such as the one shown at504 of FIG. 5 is not formed and apparent shake on an image is differentfrom translational shake.

However, as illustrated in FIG. 6, when shake of the image-pickupapparatus 601 itself is minute, approximation is possible in whichimage-pickup planes 602 and 603 before and after panning/tilting shakemove on the same plane to form a common area 604. Accordingly, accurateshake correction can be made even if both translational shake andpanning/tilting shake are corrected by translation correction method.

When rotational shake around the optical axis is occurring in additionto translational shake described above, a correction method based ontranslation and rotation transformation that is one of coordinatetransformation methods, is selected (hereinafter referred to as“translation and rotation correction method”).

Translational shake is corrected using T′ for translation correctiondescribed above. On the other hand, rotational shake is corrected in thefollowing manner. For example, when rotation of the image-pickupapparatus by an angle θ occurs around the optical axis, rotational shakeR can be expressed as: $\begin{matrix}{R = \begin{bmatrix}{\cos\quad\theta} & {{- \sin}\quad\theta} & 0 \\{\sin\quad\theta} & {\cos\quad\theta} & 0 \\0 & 0 & 1\end{bmatrix}} & ( {{Expression}\quad 3} )\end{matrix}$Thus, translation and rotation correction can be expressed as:a=R ⁻¹ a′+T′  (Expression 4)

The value of rotational shake R can be calculated from the amount ofshake between successive frame images that is sent from the shakeanalysis part 109. Translation and rotation transformation is generallycalled Euclidean transformation.

When zooming shake and shearing shake are also occurring in addition totranslational shake and rotational shake around the optical axisdescribed above, a correction method based on affine transformation thatis one of coordinate transformation methods, is selected (hereinafterreferred to as “affine correction method”). Affine transformation can beexpressed by the matrix below: $\begin{matrix}{H_{a} = \begin{bmatrix}h_{11} & h_{12} & h_{13} \\h_{21} & h_{22} & h_{23} \\0 & 0 & 1\end{bmatrix}} & ( {{Expression}\quad 5} )\end{matrix}$

Thus, assuming that point a on the image moves to point a′ due to shakeHa expressed by the affine matrix above, affine correction forcorrecting it can be expressed as:a=H_(a) ⁻¹a′  (Expression 6)

Values of elements of shake Ha expressed by the affine matrix can becalculated from the amount of shake between successive frame images thatis sent from the shake analysis part 109.

When shake involving trapezoidal deformation is occurring in addition toshake that can be expressed by affine transformation described above, acorrection method based on plane projection transformation that is oneof coordinate transformation methods, is selected (hereinafter referredto as “plane projection correction method”). Shake involving trapezoidaldeformation occurs when the amount of shake on an image is large. Insuch a situation, positions of the image-pickup planes 502 and 503before and after panning/tilting widely incline as shown in FIG. 5.Thus, approximation between translational shake and panning/tiltingshake discussed above does not hold.

Accordingly, it is necessary to select plane projection correctionmethod that can handle translational shake and panning/tilting shakeseparately. Plane projection transformation can be expressed by thematrix below: $\begin{matrix}{H_{P} = \begin{bmatrix}h_{11} & h_{12} & h_{13} \\h_{21} & h_{22} & h_{23} \\h_{31} & h_{32} & h_{33}\end{bmatrix}} & ( {{Expression}\quad 7} )\end{matrix}$

Assuming that point a on the image moves to point a′ due to shake Hpwhich is expressed by the plane projection matrix above, planeprojection correction for correcting it can be expressed as:a=H_(p) ⁻¹a′  (Expression 8)

Values of elements of shake Hp expressed by the plane projection matrixcan be calculated from the amount of shake between successive frameimages that is sent from the shake analysis part 109.

Further, when shake correction is made taking into consideration thedepth of an image-taken scene, so-called parallax correction method isselected.

Each of the methods described above from translational correction methodto plane projection correction method is a correction method thatapproximates an image-taken scene to be a plane. However, whendifference in depth of an image-taken scene is large, that is,difference in distance to subjects contained in the image-taken scene islarge, it is difficult to assume that all the subjects are on the sameplane.

That is, as illustrated in FIG. 7, consider an image-taken scene inwhich an object 702 is present near an image-pickup apparatus 701 and anobject 703 is present far from the apparatus 701. In this case, when theimage-pickup apparatus 701 moves, the amount of apparent movement of theobject 705 which is close to the image-pickup apparatus 701 becomeslarger, and, on the contrary, that of the object 706 which is far fromthe apparatus 701 becomes smaller, on a taken image 704 obtained fromthe image-pickup apparatus 701.

In such a situation, a properly shake-corrected image cannot be producedeven if only one of methods from translational correction to planeprojection correction is applied to one frame image. Thus, it isnecessary to apply shake correction processing that takes intoconsideration of difference in nature of shake that results fromdifference in depth.

In this embodiment, a method that applies segmentation to an image andclassifies subjects according to their distances from the image-pickupapparatus so that an optimal correction method is applied to therespective subjects is referred to as a parallax correction method. Theparallax correction method selects a correction method (a coordinatetransformation method) optimal for the amount of shake calculated foreach pixel, or measures the distances to subjects using a distancemeasuring sensor, e.g., an infrared sensor, to select a correctionmethod optimal for each of the distances.

When the amount of shake on an image is minute and is determined to be anegligible motion, shake correction itself may not be performed, asillustrated in FIG. 3.

The translational correction and translation and rotation correctionmethods thus far described are shake correction methods having lowdegree of freedom that assume that, when shake of the image-pickupapparatus itself is minute, shake that can be approximated to anothershake or that can be neglected is present. In this case, since only afew parameters are required for shake correction, robust and fastprocessing is possible.

However, as shake becomes larger, the above-described approximation doesnot hold and thus a correction method of low degree of freedom wouldreduce the accuracy of correction. Therefore, as shake is larger, ashake correction method of higher degree of freedom, e.g., affinecorrection and plane projection, and further parallax correctionmethods, should be selected. This can maintain high accuracy of shakecorrection.

Also, as shown in FIG. 8, in this embodiment, a correction method ofhigher degree of freedom encompasses a correction method of lower degreeof freedom. Because of this, even for an area for which parallaxcorrection method should be selected in FIG. 3, for example, translationcorrection method may be used for correction if it is determined thatonly translational shake is occurring on the image as apparent shake.

In FIG. 2, at step S204, the shake correction circuit 113 performs shakecorrection processing (coordinate transformation processing) using ashake correction method selected at step S203. The expression for theselected shake correction method is applied to each pixel of the targetimage and coordinate values after shake correction are calculated,thereby generating coordinate value transformation data for shakecorrection. Then, based on the generated coordinate value transformationdata, pixel values are read out from the image stored and held in theframe memory 107, and an image that is formed by those pixel values istransmitted to the video output circuit 114 as a shake-corrected image.

At step S205, the shake-corrected image is output from the video outputcircuit 114.

As has been thus described, this embodiment selects an optimal shakecorrection method from among the plural shake correction methods byutilizing the fact that characteristics of a motion vector, whichrepresents shake occurring on an image, vary with the amount of shake.This can realize electronic image stabilization that can perform propershake correction despite change in characteristics of shake of animage-pickup apparatus, i.e., image-taking conditions.

Embodiment 2

FIG. 9 shows the configuration of an image-pickup apparatus thatincludes an image processing apparatus as the second embodiment of thepresent invention. This embodiment selects an optimal shake correctionmethod based on the amount of shake on an image and the angle of view ofan output image (as well as the angle of view of an input image).

In FIG. 9, components common to those shown in FIG. 1 are denoted withthe same reference numerals as in FIG. 1. In addition to theconfiguration shown in FIG. 1, the image-pickup apparatus of theembodiment includes an input angle-of-view detection circuit 912 fordetecting the angle of view of an input image (hereinafter “an inputangle of view”) that is formed by the image forming circuit 103. Theimage-pickup apparatus also includes an output angle-of-view calculationcircuit (output area determination part) 913 for calculating the angleof view for outputting an image (an output image area, hereinafterreferred to as an “output angle of view”) which has been subjected toshake correction at the shake correction circuit 915, using the inputangle of view and the amount of shake calculated by the shake analysispart 109. The main controller illustrated in FIG. 1 is omitted in FIG.9.

The operation of the image-pickup apparatus (image processing apparatus)of the embodiment will be described below with respect to the flowchartshown in FIG. 10.

Steps S1001 and S1002 are similar to steps S201 and S202 shown in thefirst embodiment (FIG. 2), respectively.

At S1003, the output angle-of-view calculation circuit 913 calculatesthe output angle of view in advance that is used when theshake-corrected image is sent to the video output circuit 114. Theoutput angle of view is calculated using the input angle of viewdetected by the input angle-of-view detection circuit 912 and the amountof shake calculated by the shake analysis part 109. The input angle ofview is calculated based on positional information for lensesconstituting the optical system 101 and focal length information of theoptical system 101. The calculated output angle of view is sent to thecorrection method selection circuit 914 and the shake correction circuit915.

At step S1004, using the amount of shake provided by the shake analysispart 109 and the output angle of view provided by the outputangle-of-view calculation circuit 913, shake that is occurring in theimage currently being taken is analyzed. And a shake correction methodoptimal for correcting the shake is selected.

In this embodiment, an area of an input image that will be actuallyoutput is selected as a take-out area of an output image. Then, using anoptimal shake correction method, shake correction is performed by movingand deforming the take-out area within the input image depending on theshake. Use of such a method can prevent missing of pixels in an outputimage due to shake correction.

The size of a take-out area, that is, the angle of view of an outputimage, can be determined from its proportion to the size of an inputangle of view, for example. Alternatively, it is also possible toanalyze the amount of apparent shake appearing on an image to calculatean output angle of view at which a take-out area does not lie outsidethe range of an input image even if shake correction is applied, forexample.

In a case where the size of an output angle of view is selected so thatbetter shake correction can be made, when the size of the output angleof view is changed, characteristics of apparent shake appearing on theimage also change.

For instance, as illustrated in FIG. 11, when the proportion of theangle of view of an output image 1102 is large relative to the angle ofview of an input image 1101, the range in which a take-out area of theoutput image can move is small. Thus, only small shake can be corrected.

In contrast, when the proportion of the angle of view of an output image1202 is small relative to the angle of view of an input image 1201 asshown in FIG. 12, the range in which a take-out area of the output imagecan move is large. Thus, large shake can be also corrected.

If there is a motion having such a magnitude that would cause a take-outarea to lie outside the range of the input image if correction isperformed, the motion is determined to be uncorrectable shake or anintentional camera work and excluded from shake correction.

Now, description will be given of how to select a shake correctionmethod in this embodiment. FIG. 13 shows the relation among magnitude ofshake, the size of an output angle of view, and a corresponding shakecorrection method optimal for them.

As seen from the figure, when an output angle of view is set to belarge, the amount of correctable shake appearing on the image is small,so that small shake of the image-pickup apparatus itself should becorrected. Thus, a shake correction method of low degree of freedom,such as translation correction and translation and rotation correctionmethods, is selected. This enables robust and fast shake correction. Onthe other hand, when the output angle of view is set to be small, theamount of correctable shake appearing on an image is large, so thatlarge shake of the image-pickup apparatus itself should be corrected.Thus, a shake correction method of high degree of freedom, such asaffine correction, plane projection correction, and further parallaxcorrection methods, is selected. This can realize accurate shakecorrection that also addresses complicated shake.

In FIG. 10, at step S1005, the shake correction circuit 915 appliesshake correction processing using the shake correction method selectedat step S1004. The expression for the selected shake correction methodis applied to each pixel of the target image and coordinate values aftershake correction are calculated to generate coordinate valuetransformation data for shake correction. Based on the generatedcoordinate value transformation data and the size of the output angle ofview calculated in advance by the output angle-of-view calculationcircuit 913, pixel values are taken out from the image stored and heldin the frame memory 107. Then, an image that is formed by those pixelvalues is sent to the video output circuit 916 as a shake-correctedimage.

At step S1006, the shake-corrected image is output from the video outputcircuit 114.

As has been thus described, according to the embodiment, an optimalshake correction method is selected by utilizing the fact thatcharacteristics of a motion vector that represents shake appearing on animage vary with the amount of shake and the size of an output angle ofview. This can realize electronic image stabilization that can performproper shake correction despite change in shake characteristics of animage-pickup apparatus.

Embodiment 3

FIG. 14 shows the configuration of an image-pickup apparatus thatincludes an image processing apparatus as the third embodiment of thepresent invention. This embodiment has a zoom optical system withvariable focal length and selects an optimal shake correction methodaccording to the amount of shake on an image and the zooming position(i.e. focal length) of the zoom optical system.

In FIG. 14, components common to those shown in FIG. 1 are denoted withthe same reference numerals as in FIG. 1. The image-pickup apparatus ofthe embodiment includes a zoom position control circuit 1412 forcontrolling the zoom position of the zoom optical system 1401 and a zoomposition detection circuit 1413 for detecting the zoom position, inaddition to the configuration shown in FIG. 1. The main controller shownin FIG. 1 is omitted in FIG. 9.

The operation of the image-pickup apparatus (image processing apparatus)of the embodiment will be described below with the flowchart shown inFIG. 15.

Steps S1501 and S1502 are similar to steps S201 and S202 shown in thefirst embodiment (FIG. 2), respectively.

At step S1503, the zoom position of the zoom optical system 1401 iscontrolled by the zoom position control circuit 1412 in accordance withoperation of a zoom switch, not shown, by a user. Also, the zoomposition is detected by the zoom position detection circuit 1413.Detected zoom position information is sent to the correction methodselection circuit 1414.

Here, description will be given of how to selects a shake correctionmethod in this embodiment. When the zoom position of the zoom opticalsystem 1401 is changed, characteristics of apparent shake on an imagechange. Shake that appears on an image results from shake of theimage-pickup apparatus itself, and when the zoom position is moved totelephoto side for example, the displayed image enlarges and the amountof apparent shake on the image also increases. Thus, even when shake ofthe same size is occurring both on a telephoto image and a wide-angleimage, the magnitude of actual shake of the image-pickup apparatusitself is smaller when the apparatus is taking the telephoto image.

That there is difference in the magnitude of shake of the image-pickupapparatus itself means that there is difference in characteristics ofapparent shake appearing on an image as well. Thus, by analyzing shakecharacteristics using the amount of shake and the zoom positioninformation, it is possible to estimate the magnitude of shake of theimage-pickup apparatus itself and determine a shake correction method ofhow much degree of freedom should be applied.

FIG. 16 shows the relation between the amount of shake and the zoomposition of the zoom optical system 1401, and shake correction methodsoptimal for them in this embodiment.

As seen from the figure, when the zoom position is telephoto side andthe amount of shake is small, the amount of shake remains small despiteenlarged shake due to telephoto image-taking. Thus, shake of theimage-pickup apparatus itself is estimated to be minute. It follows thatapproximation between translational shake and panning/tilting shake doeshold and a correction method of low degree of freedom can be selected,e.g., translation correction and translation and rotation correctionmethods. This enables robust and fast shake correction processing.

On the other hand, as the zoom position moves to wide-angle side or theamount of shake is larger, shake of the image-pickup apparatus itselfbecomes larger. Thus, the above-described approximation does not holdand shake including more complicated motion occurs. For such shake, moreaccurate shake correction can be realized by selecting a shakecorrection method of higher degree of freedom such as affine correction,plane projection correction, and further parallax correction methods.

In FIG. 15, at step S1505, the shake correction circuit 1415 appliesshake correction processing using a shake correction method selected atstep S1504. The expression for the selected shake correction method isapplied to each pixel of the target image and coordinate values aftershake correction are calculated to generate coordinate valuetransformation data for shake correction. Then, based on the generatedcoordinate value transformation data, pixel values are read out from animage stored and held in the frame memory 107 and an image that isformed by those pixel values is sent to the video output circuit 114 asa shake-corrected image.

At step S1506, the shake-corrected image is output from the video outputcircuit 114.

As has been thus described, this embodiment selects an optimal shakecorrection method by utilizing the fact that characteristics of a motionvector that represents shake appearing on an image vary with the amountof shake and/or the zoom position. This can realize electronic imagestabilization that can perform proper shake correction even if shakecharacteristics of the image-pickup apparatus change.

Embodiment 4

FIG. 17 shows the configuration of an image-pickup apparatus thatincludes an image processing apparatus as the fourth embodiment of thepresent invention. This embodiment selects an optimal shake correctionmethod based on the amount of shake on an image, the zoom position ofthe zoom optical system, and the angle of view of an output image (aswell as that of an input image (an input angle of view)).

In FIG. 17, components common to those shown in FIG. 1 are denoted withthe same reference numerals as in FIG. 1. The image-pickup apparatus ofthe embodiment has the input angle-of-view detection circuit 912 and theoutput angle-of-view calculation circuit 913 which have been describedin the second embodiment, in addition to the configuration shown inFIG. 1. The image-pickup apparatus of the embodiment further has thezoom position control circuit 1412 and the zoom position detectioncircuit 1413 which have been described in the third embodiment.

The operation of the image-pickup apparatus (or image processingapparatus) of the embodiment will be described below with the flowchartshown in FIG. 18.

Steps S1801 and S1802 are similar to steps S201 and S202 shown in thefirst embodiment (FIG. 2), respectively.

At step S1803, the zoom position control circuit 1412 controls the zoomposition of the zoom optical system 1401 in accordance with operation ofa zoom switch, not shown, by a user. Also, the zoom position detectioncircuit 1413 detects the zoom position. Detected zoom positioninformation is sent to the correction method selection circuit 1716.

At step S1804, the output angle-of-view calculation circuit 913calculates in advance an output angle of view for sending ashake-corrected image to the video output circuit 114. The output angleof view is calculated using an input angle of view detected by the inputangle-of-view detection circuit 912 and the amount of shake calculatedby the shake analysis part 109. The input angle of view is calculatedbased on position information for a variable magnification lens unitconstituting a part of the zoom optical system 1401 and focal lengthinformation of the zoom optical system 1401. The calculated output angleof view is sent to the correction method selection circuit 1716 and theshake correction circuit 1717.

At step S1805, shake occurring in the image which is currently beingtaken is analyzed using the amount of shake provided by the shakeanalysis part 109, the zoom position provided by the zoom positiondetection circuit 1413, and the output angle of view provided by theoutput angle-of-view calculation circuit 915. Then, a shake correctionmethod most appropriate for correcting the shake is selected.

Here, description will be given of how to select a shake correctionmethod in this embodiment. The shake correction in this embodiment isperformed by analyzing characteristics of shake that appears on an imageand then selecting a method best suited for correcting the shake fromamong plural shake correction methods having different degrees offreedom so as to apply shake correction processing.

Also, shake as called in this embodiment refers to shake of theimage-pickup apparatus itself appearing as image motion on a takenimage.

This embodiment analyzes shake using values such as the amount of shakeon an image, the zoom position, and the size of the output angle of viewto estimate condition of shake of the image-pickup apparatus itself.And, based on information contained in it such as the type and themagnitude of shake and availability of approximation, this embodimentselects a shake correction method having the most appropriate degree offreedom.

The relation among the amount of shake, the output angle of view, andthe shake correction method to be selected in this embodiment is similarto the one described in the second embodiment (FIG. 13). The relationamong the amount of shake, the zoom position and the shake correctionmethod to be selected is similar to the one described in the thirdembodiment (FIG. 16).

FIG. 19 illustrates the relation among the size of the output angle ofview, the zoom position and the shake correction method that should beselected. In the figure, when the output angle of view is large, therange in which correction is possible in an input image is small, sothat correction is applied to small shake only. Consequently, it ispossible to select a correction method of low degree of freedom , suchas translation correction, and translation and rotation correctionmethods.

In contrast, when the output angle of view is small, the range in whichcorrection is possible in an input image is large and thus shake to becorrected is larger and more complicated. Thus, it is necessary toselect a shake correction method of higher degree of freedom, such asaffine correction, plane projection correction, and further parallaxcorrection methods.

When the zoom position is telephoto side, small shake of theimage-pickup apparatus itself appears being enlarged as shake on animage, so that a correction method may be selected according to the sizeof the output angle of view as described above. However, when the zoomposition is wide-angle side, actual shake of the image-pickup apparatusitself becomes large, so that it is necessary to select a shakecorrection method of high degree of freedom as appropriate asillustrated in FIG. 19.

In this manner, a shake correction method that has the most appropriatedegree of freedom for correcting shake occurring on an image is selectedbased on the relation among the amount of shake, the zoom position, andthe output angle of view.

In FIG. 18, at step S1806, the shake correction circuit 1717 appliesshake correction processing using the shake correction method selectedat step S1805. Shake correction processing is performed in a similar wayto steps S1005 and S1505 in the second and third embodiments describedabove.

At step S1807, a shake-corrected image is output from the video outputcircuit 114.

As has been described above, this embodiment selects an optimal shakecorrection method by utilizing the fact that characteristics of a motionvector that represents shake appearing on an image vary with the amountof shake, the zoom position, and size of the output angle of view. Thiscan realize electronic image stabilization that can perform proper shakecorrection despite change in shake characteristics of an image-pickupapparatus.

As has been described, according to the embodiments described above, itis possible to select a coordinate transformation method for use incoordinate transformation processing (image processing) which isperformed as shake correction processing optimally in variousimage-taking conditions, that is, conditions of shake occurrence. Thus,it is possible to realize electronic image stabilization function thatcan provide a good output image (i.e., a shake-corrected image)irrespective of image-taking conditions.

While each of the above embodiments described a case where thecorrection method selection circuit selects a shake correction method tobe applied based on information such as the amount of shake, the presentinvention is not limited thereto. For example, a user may be allowed toarbitrarily select a shake correction method which the user considers tobe best suited for a image-taking condition by operating an operationmember, e.g., a switch.

In addition, while each of the above embodiments described a case wherethe image-pickup apparatus contains an image processing apparatus thathas shake correction function, the present invention is not limitedthereto. For example, as illustrated in FIG. 20, an image taken by theimage-pickup apparatus 250 (an input image) may be transmitted to apersonal computer 260. The transmission may be made by cable orwirelessly, or via the Internet or a LAN. Then, selection of a shakecorrection method and shake correction processing, which are illustratedin the flowcharts of FIGS. 2, 10, 15, and 18, can be performed on thepersonal computer 260. In this case, the personal computer 260 functionsas the image processing apparatus of the invention.

Further, in this case, the amount of shake (i.e., motion vectors) may bedetected by the personal computer 260 or an output from a shake sensorcontained in the image-pickup apparatus may be supplied to the personalcomputer 260.

Furthermore, the present invention is not limited to these preferredembodiments and various variations and modifications may be made withoutdeparting from the scope of the present invention.

This application claims foreign priority benefits based on JapanesePatent Application No. 2005-360108, filed on Dec. 14, 2005, which ishereby incorporated by reference herein in its entirety as if fully setforth herein.

1. An image processing apparatus, comprising: a shake correcting partthat performs coordinate transformation processing based on shakeinformation to an input image that is generated by use of animage-pickup device; and a method changing part that changes acoordinate transformation method for the coordinate transformationprocessing.
 2. The image processing apparatus according to claim 1,wherein the method changing part changes the coordinate transformationmethod based on the shake information.
 3. The image processing apparatusaccording to claim 2, further comprising an output area determining partthat determines an output image area within the input image based onangle of view information for the input image and the shake information,wherein the method changing part changes the coordinate transformationmethod based on the shake information and information on the outputimage area.
 4. The image processing apparatus according to claim 2,wherein the method changing part changes the coordinate transformationmethod based on the shake information and zoom information for a zoomoptical system that forms an image of a subject on the image-pickupdevice.
 5. The image processing apparatus according to claim 2, furthercomprising an output area determining part that determines an outputimage area within the input image based on angle of view information forthe input image and the shake information, wherein the method changingpart determines a coordinate transformation method based on the shakeinformation, zoom information for a zoom optical system that forms animage of a subject on the image-pickup device, and information on theoutput image area.
 6. The image processing apparatus according to claim1, further comprising a shake information generating part that generatesthe shake information from the input image.
 7. The image processingapparatus according to claim 1, wherein the method changing part changesthe coordinate transformation method between at least two of translationtransformation method, translation and rotation transformation method,affine transformation method, and plane projection transformationmethod.
 8. The image processing apparatus according to claim 7, whereinthe method changing part changes a coordinate transformation method foreach of areas that have different distances in the input image.
 9. Theimage processing apparatus according to claim 1, wherein the methodchanging part changes the coordinate transformation method in accordancewith operation of the image processing apparatus by a user.
 10. Animage-pickup apparatus, comprising: an image-pickup device; and theimage processing apparatus according to claim
 1. 11. An image processingmethod, comprising the steps of: obtaining shake information; performingcoordinate transformation processing based on the shake information toan input image that is generated by use of an image-pickup device; andchanging a coordinate transformation method for the coordinatetransformation processing.